Add mxq rolling reboot #558
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reboot mxqd cluster nodes into a new kernel without killing running jobs and without draining the whole cluster at once. For each outdated node it drains mxqd (stop, then wait for the mxqd process to exit), installs a one-time systemd unit that runs mxqdctl-hostconfig start on the next boot and removes itself again, selects the kernel with mxgrub and reboots (kexec by default, or a normal reboot for firmware changes), all detached via systemd-run so long drains survive an SSH disconnect. Pacing is load-aware from the mxq server status page: at most BUSY_MAX nodes (default 2) when the cluster is fully busy, otherwise up to half of the currently idle nodes. Nodes already on the target kernel are detected via uname -r and skipped, so re-running is safe. The interactive (mxqi) nodes are handled as a separate, throttled pool: at most MXQI_MAX of them (default 1) are upgraded at a time and at least MXQI_MIN_UP (default 1) is always kept up and serving, so interactive sessions can always be started. mxqi nodes are auto-detected from /etc/hostconfig as the mxqd nodes whose prerequisites request 'mxqi'. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ocking The mxqi keep-serving guard counted only nodes serving on the status page and then assumed the node about to be taken down was one of them (serving - 1 >= MXQI_MIN_UP). When every mxqi node already had mxqd stopped or draining, none appeared on the page, so 'serving' was 0 and the guard held forever, waiting for a node that could not return on its own. Only protect nodes that are actually serving: prefer an mxqi node that is already down (rebooting it removes no interactive capacity), and apply the keep-serving throttle only when the remaining candidates are still serving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the HTML scrape of the mxq server status page with a direct query of the mxq_daemon table via /etc/mxq/mysql_ro.cnf. This is a machine-readable, authoritative source and drops the curl dependency. Take the newest daemon row per host and keep only daemons that are actually accepting jobs (status 0 = idle, 10 = running); a draining daemon (status 200) or a gone one (>= 250) is left out, so the 'serving' notion the mxqi keep-serving guard relies on stays correct. Rows not refreshed within DB_FRESH_MIN minutes are ignored so a daemon that vanished without recording a stop is not mistaken for live. Skip a poll when the query fails instead of acting on an empty result, which would otherwise look like 'nothing is serving' and could wrongly free the mxqi guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tool to reboot mxq cluster nodes into new Linux kernel: